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CLAIMS 

1 1. A computer-implemented method for analyzing trace information 

2 generated during execution of multiple threads of a software program on a first computer, 

3 the first computer having multiple processors that each have multiple protection domaitis 

4 that are each able to execute at least one of the multiple threads, each processor having a 

5 counter indicating a number of instruction holes during which an instruction is not 

6 executed by the processor, each protection domain having a counter indicating a number 

7 of instiiictions issued in the protection domain by all executing threads, the method 

8 comprising: 

9 receiving an indication of trace information reflecting a series of events that 

10 occurred during the execution, each event associated with execution of one of the 

1 1 multiple threads by one of the protection domains of one of tiie processors and each event 

12 having associated values in the trace information of variables maintained by the executing 

13 software program, by the one protection domain, and/or by the one processor; 

14 for each of a plurality of periods of time during which the execution was 

15 occurring, 

16 determining from the trace information a number of instructions 

17 executed for the software program during the period of time by 

identifying multiple protection domains that each executed at 

1 9 least one of the multiple threads during at least a portion of the period of time; 

20 for each of the identified protection domains, 

21 determining a change in the value of the issued 

22 instructions counter of the protection domain during the period of time ; 

23 determining if all of the instructions issued in the 

24 protection domain during the period of time were for one of the multiple threads; 

25 when it is determined that all of the instructions issued 

26 in the protection domain during the period of time were for one of the multiple threads, 
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27 calculating a value for the number of instructions executed for the software program 

28 during the period of time by the protection domain to be the determined change; and 

29 when it is determined that all of the instructions issued 

30 in the protection domain during the period of time were not for one of the multiple 

31 threads, calculating a value for the number of instructions executed for the software 

32 program during the period of time by the protection domain to be a portion of the 

33 determined change that corresponds to a portion of the period of time during which at 

34 least one thread for the software program was executing in the protection domain; and 

35 determining the number of instructions executed for the 
:B6 software program during the period of time to be a sum of the calculated values for each 
;i37 of the identified protection domains; and 

^138 determining from the trace information a number of instruction slots 

N39 available for execution of the instructions of software program during the period of time 

mo by 

,,41 identifying processors that each executed at least one of the 

F42 multiple threads during the period of time ; 

Ei43 for each of the identified processors, 

744 determining a change in the value of the instruction 

45 holes counter of the processor during the period of time; and 

46 if all of the instruction holes that occurred during the 

47 period of time were attributable to the software program, calculating a value for the 

48 number of instruction holes for the processor that are attributable to the software program 

49 during the period of time to be the determined change in the value of the instruction holes 

50 counter; 

51 calculating a value for the nimiber of instruction holes tiiat are 

52 attributable to the software program during the period of time by all of the identified 

53 processors to be a sum of the calculated values for each of the identified processors; and 

54 determining the number of instruction slots available for 
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execution of the instructions of software program during the period of time to be a sum of 
the determined number of instructions executed for the software program during the 
period of time and of the calculated value for the number of instruction holes that are 
attributable to the software program during the period of time; and 

presenting to a user an indication of the determined number of executed 
instructions for each of the periods of tune and an indication of the determined number of 
available instruction slots for each of the periods of time. 

2. The method of claun 1 wherein only one software program can 
execute in a protection domain at any point in time, and wherein when it is determined 
that all of the instructions issued in a protection domain during a period of time were not 
for one of the multiple threads, the calculating of a value for the number of mstructions 
executed for the software program during the period of time by the protection domain 
includes: 

determining from the trace information at least one swap event that 
occurred in the protection domain during the period of time such that the software 
program is swapped into the protection domain so as to commence execution of the 
software program or such that the software program is swapped out of the protection 
domain so as to suspend execution of the software program; 

retrieving for each of the determined swap events an associated value in the 
trace information of the issued instructions counter of the protection domain; and 

using the retrieved associated values to calculate the value for the number 
of instructions executed for the software program during the period of time by the 
protection domain to include only increments to the issued instructions counter that 
occurred while the soflwaxe program is swapped into the protection domain. 

3. The method of claim 1 wherem, for at least one of the identified 
protection domains for at least one of the periods of time, there are no variable values in 
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3 the trace information indicating a value for the issued instructions counter of that 

4 protection domain at an end of that period of time, and wherein the determining of a 

5 second value for the issued instructions counter of that protection domain at the end of 

6 that period of time includes estimating the second value based on an extrapolation 

7 between earher and later values for that issued instructions counter. 

1 4. The method of claim 1 wherein, for at least one of the identified 

2 processors for at least one of the periods of time, there are no variable values in the trace 

3 information indicating a value for the instruction holes counter of that processor at an end 

4 of that period of time, and wherein the determining of a second value for the instruction 

5 holes counter of that processor at the end of that period of time includes estimating the 

6 second value based on an extrapolation between earlier and later values for that 

7 instruction holes counter. 

1 5. The method of claim 1 wherein, for at least one of the identified 

2 protection domains for at least one of the periods of time, no event occurred during the 

3 execution of the software program for that protection domain for that period of time, and 

4 including estimating the first and second values for the issued instructions counter of that 

5 protection domain. 

1 6. The method of claim 1 wherein, for at least one of tiie identified 

2 processors for at least one period of time, at least one other software program is executing 

3 during that period of time, and including, for each of the at least one identified 

4 processors: 

5 when it is determined that all of the instruction holes that occurred during 

6 that period of time were not attiibutable to the software program, 

7 calculating a value for the number of instruction holes for that 

8 processor that are attributable to the at least one other software programs during that 
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period of time; and 

calculating a value for the number of instruction holes for that 
processor that are attributable to the software program during that period of time to be a 
difference of a total number of instruction holes for that processor during that period of 
time and the calculated value for the number of instruction holes for that processor that 
are attributable to the at least one other software programs. 

7. The method of claim 6 wherein the at least one other software 
programs include only an operating system program, and wherein the calculated value for 
the number of instruction holes for each processor that are attributable to the operating 
system are zero. 

8. The method of claim 1 wherein at least one of processors performing 
the execution has multiple streams performing the execution such that each of the 
multiple streams executes at least one of the threads, and including displaying 
information about the streams. 

9. The method of claim 1 wherein the presenting to the user of the 
indication of the determined number of executed instructions for each of the periods of 
time and of the indication of the determined number of available instruction slots for each 
of the periods of time includes displaying a graph including the indications. 

10. The method of claim 9 wherein the displayed indication of the 
determined number of available instruction slots for each period of time includes a 
displayed indication of the calculated number of instruction holes that are attributable to 
the software program during the period of time, with the calculated number of instruction 
holes displayed in such a manner that a user can visually aggregate the displayed 
indication of the determined number of executed instructions for that period of time with 
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the displayed indication of calculated number of instraction holes for that period of time. 

1 1 . The method of claim 9 wherein the displayed graph includes a time- 
based axis, and wherein the displayed indications of the determined number of executed 
instructions and the determined number of available instruction slots for each of the 
periods of time are points on the graph. 

12. The method of clarni 11 wherein the displayed graph includes an 
origin with at least two axes, and including, after the displaying of the indications of the 
determined number of executed instructions and of the determined number of available 
instruction slots, redefining at least one of the axes based on a new indicated displayed 
location. 

13. The method of claim 1 including, for at least one of the periods of 
time, presenting an indication of a logical code block of the software program that was 
executing during that period of time. 

14. The method of claim 13 wherein the presented indication of the 
logical code block is a name of the logical code block. 

15. The method of claim 13 wherein the presented indication of the 
logical code block is source code of the logical code block. 

16. The method of claim 13 wherein the logical code block is a fimction. 

17. The method of claim 1 including presenting at least some of the 
variable values from the trace information in a tabular format. 



50 



1 18. The method of claim 1 wherein the number of processors identified 

2 during each of the periods of time is greater than one, and wherein the determined 

3 number of available instruction slots for each of the periods of time is the identified 

4 number of processors for that period of time. 

1 19. The method of claim 1 wherein a fnst number of processors 

2 identified during a fnst period of time is distinct from a second number of processors 

3 identified during a second period of time. 

! 1 20. The method of claim 1 wherein the identified number of processors 

%i for at least one of the periods of time is greater than one, and wherein information for 

^. 3 each of the processors is aggregated during the presenting of information for those 

- 4 periods of time. 

f : 1 21. The method of claim 20 including normahzing the information for 

^ 2 each of the processors prior to the presenting of the information by determining a value 

i 3 for the instruction holes counter of each of the processors at a beginning of the execution 

4 of the software program and by subtracting the determined value for each instruction hole 

5 counter from each later value of that instruction hole counter. 

1 22. The method of claim 1 wherein information to be presented is 

2 determined based on a specification provided by a user. 

1 23. The method of claim 22 wherein the specification is used at the time 

2 of the presenting to determine the information to be presented. 

1 24. The method of claim 1 including presenting information about 
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2 memory references performed during at least one of the periods of time. 

1 25. The method of claim 1 including presenting information about 

2 FLOPS performed during at least one of the periods of time. 

1 26. The method of claim 1 including presenting information about a 

2 number of the threads in existence during at least one of the periods of time. 

1 27. The method of claim 1 including presenting information about a 

2 number of the threads that are blocked during at least one of the periods of time. 

1 28. The method of claim 1 including presenting information about a 

2 number of the threads that are ready for execution durmg at least one of the periods of 

3 time. 

1 29. The method of claim 1 including presenting information about 

2 contention for locks during at least one of the periods of time. 

1 30. The method of claim 1 wherein the presenting of information is in 

2 response to a request by a user. 

1 31. The method of claim 1 wherein the presenting of information is 

2 performed automatically without user intervention. 

1 32. The method of claim 1 including generating the indicated trace 

2 information. 

1 33. The method of claim 1 wherein the software program is an 
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executable version of a source code program such that execution of the executable 
version will generate the mdicated trace information. 

34. The method of claim 33 including generating the executable version 
by compiling the source code program such that a group of instructions is added to the 
source code program at a location specified by the user, the added instructions when 
executed to generate trace information for a type of event specified by the user. 

35. The method of claim 33 including generating the executable version 
by compiling the source code program such that groups of instructions are added to the 
source code program at a plurahty of automatically identified locations of the software, 
the added instructions when executed to generate trace information of a specified type. 

36. The method of claim 35 wherein the automatically identified 
locations include beginnings of functions having more lines than a threshold. 

37. The method of claim 35 wherein the automatically identified 
locations include ends of fimctions having more lines than a threshold. 

38. The method of claim 35 wherein the automatically identified 
locations include locations where a compiler adds instructions related to parallelizing the 
execution of the executable version. 

39. The method of claim 38 wherein the instructions related to 
parallelizing the execution of the executable version are instructions related to a fork. 

40. The method of claim 38 wherein the instructions related to 
parallelizing the execution of the executable version are instructions related to a joiti. 
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41. The method of claim 34 wherein, for at least some of the groups of 
instructions to be added to the executable program to generate trace information of a 
specified type, an additional group of instructions is added to the source code program 
that when executed create a descriptor object for that group of instructions. 

42. The method of claim 41 wherein the additional groups of 
instructions are added to the source code program in such a manner that the additional 
groups of instructions will be executed before any of the groups of instructions. 

43. The method of claim 1 including generating the indicated trace 
information by executing the executable version. 

44. The method of claim 43 wherein, during the executing of the 
executable version, execution of each of the added groups of instructions adds current 
values of one or more of the variables to the generated trace information. 

45. The method of claim 43 wherein, during the executing of the 
executable version and before the execution of an added group of instructions, an 
additional group of added instructions is executed and creates a descriptor object for that 
added group of instructions. 

46. The method of claims 43 wherein information is added to the 
generated indicated trace information by an executing program other than the software 
program. 

47. The method of claim 46 wherein the other executing program is an 
operating system. 
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1 48. The method of claim 46 wherein the other executing program is a 

2 background daemon. 

1 49. The method of claim 1 including analyzing the trace information to 

2 extract execution information corresponding to the events that occurred during the 

3 execution. 

1 50. The method of claim 49 wherein the analyzing of the tiace 

2 information includes using a description of specified types of events to identify groups of 

3 trace information each reflecting execution of a group of added instructions. 

1 51. The method of claim 50 wherein the analyzing of the trace 

2 information includes, when information for an event in the tiace information is separated 

3 in multiple non-contiguous portions, creating a mapping to assist in retrieving the 

4 information for the event. 

1 52. The method of claim 51 including using the identified groups of 

2 trace information and the created mapping to extract current values for execution 

3 information. 

1 53. The method of claim 49 wherein tiie analyzing of the tiace 

2 information includes modifying current values of events so as to be normalized with 

3 respect to begiiming of the execution. 

1 54. The metiiod of claim 49 wherein the analyzing of the tiace 

2 information includes modifying current values of events so that the values reflect only tiie 

3 execution of the multiple threads. 
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1 55, The method of claim 49 including using information retrieved from 

2 created descriptor objects in the trace information. 

1 56. A computer system for analyzing execution of multiple threads of a 

2 software program on a first computer, the first computer having multiple processors that 

3 each have multiple protection domains that are each able to execute at least one of the 

4 multiple threads, each processor having a counter indicating a number of instruction slot 

5 holes during which an instruction is not executed by the processor, each protection 

6 domain having a counter indicating a number of instructions executed in the protection 

7 domain by all executing threads, comprising: 



8 means for analyzing that 

9 receives an indication of trace information reflecting a series of 

10 events that occurred during the execution, each event associated with execution of one of 

11 the multiple threads by one of the protection domains of one of the processors and each 

12 event having associated values in the trace information of variables maintamed by the 

13 executing software program, by the one protection domam, and/or by the one processor; 

14 for each of a plurality of periods of time during which the execution 

15 was occurring, determines from the trace information a number of instructions executed 

16 for the software program during the period of time by 

17 identifyiag multiple protection domains that each executed at 



18 least one of the multiple threads during at least a portion of the period of time; 

19 calculating for each of the identified protection domains a 

20 value for the number of instructions executed for the software program during the period 

21 of time by the protection domain to be a portion of a change in the executed instructions 

22 counter of the protection domain during the period of time such that the portion of the 

23 change corresponds to a portion of the period of time during which at least one thread for 

24 the software program was executing in the protection domain; and 




56 



25 determining the number of instructions executed for the 

26 software program during the period of time to be a sum of the calculated values for each 

27 of the identified protection domains; and 

28 for each of the plurality of periods of time during which the 

29 execution was occurring, determines from the trace information a number of instruction 

30 slots available for execution of the instructions of software program during the period of 

31 time by 

32 identifying processors that each executed at least one of the 

33 multiple threads during the period of time; 

1:34 calculating for each of the identified processors a value for 



ms the number of instruction slot holes for the processor that are attributable to the software 

i;';36 program during the period of time to be a portion of a change in the instruction slot holes 

'^31 counter of the processor during the period of time such that other portions of the change 

:p38 corresponding to execution of other software programs by the processor are not mchided 

p^39 in the portion; and 



;F40 determining the number of instruction slots available for 

I mi execution of the instructions of software program during the period of time to be a sum of 

il42 the determined number of instructions executed for the software program during the 

43 period of time and of a sum of the calculated values for each of the identified processors 

44 for the period of time; and 

45 means for displaying an indication of the determined number of executed 

46 instructions for each of the periods of time and an indication of the determined number of 

47 available instruction slots for each of the periods of time, 

1 57. A computing device for analyzing execution of multiple threads of a 

2 software program on a first computer, the fnst computer having multiple processors that 

3 each have multiple protection domains that are each able to execute at least one of the 

4 multiple threads, each processor having a counter indicating a mraiber of instruction slot 
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5 holes during which an instruction is not executed by the processor, each protection 

6 domain having a counter indicating a number of instructions executed in the protection 

7 domain by all executing threads, comprising: 

8 a trace information receiver component capable of receiving an indication 

9 of trace information reflecting a series of events that occurred during the execution, each 

10 event associated v^ith execution of one of the multiple threads by one of the protection 

11 domains of one of the processors and each event having associated values in the trace 

12 information of variables mauitained by the executing software program, by the one 

13 protection domain, and/or by the one processor; 



^ i3i4 a trace information analysis component capable of, for each of a plurality of 

a!; 15 periods of time during which the execution was occurring, 

)'^^6 determining from the trace information a number of histructions 

; :i7 executed for the software program during the period of time by 

18 identifying multiple protection domains that each executed at 

r; 19 least one of the multiple threads during at least a portion of the period of time; 

£ 20 for each of the identified protection domains, 

^^ 21 determining a change in the value of the issued 

1=^ 22 instructions counter of the protection domain during the period of time; 

23 determining if all of the instructions issued in the 

24 protection domain during the period of time were for one of the multiple threads; 

25 when it is determined that all of the instructions issued 

26 in the protection domain during the period of time were for one of the multiple threads, 

27 calculating a value for the number of instructions executed for tiie software program 

28 during the period of time by the protection domain to be the determined change; and 

29 when it is determined that all of the instructions issued 

30 in the protection domain during the period of time were not for one of the multiple 

31 threads, calculating a value for the number of instructions executed for the software 

32 program during the period of time by the protection domam to be a portion of the 
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33 determined change that corresponds to a portion of the period of time during which at 

34 least one thread for the software program was executing in the protection domain; and 

35 determining the number of instructions executed for the 

36 software program during the period of time to be a sum of the calculated values for each 

37 of the identified protection domains; and 

38 determining from the trace information a number of instruction slots 

39 available for execution of the instructions of software program during the period of time 

40 by 

41 identifying processors that each executed at least one of the 
' t42 multiple threads during the period of time; 

;ri43 for each of the identified processors, 

1^44 determining a change in the value of the instruction 

45 holes counter of the processor during the period of time; and 
p 46 if all of the instruction holes that occurred during the 

QAi period of time were attributable to the software program, calculating a value for the 

%^ number of instruction holes for the processor that are attributable to the software program 

^ : '49 during the period of time to be the determined change in the value of the instruction holes 

^==^50 counter; 

51 calculating a value for the number of instruction holes that are 

52 attributable to the software program during the period of time by all of the identified 

53 processors to be a sum of the calculated values for each of the identified processors; and 

54 determining the number of instruction slots available for 

55 execution of the instructions of software program during the period of time to be a sum of 

56 the determined number of instructions executed for the software program during the 

57 period of time and of the calculated value for the number of instruction holes that are 

58 attributable to the software program during the period of time; and 

59 a presentation component capable of presenting to a user an indication of 

60 the determined number of executed instructions for each of the periods of time and an 
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61 indication of the determined number of available instruction slots for each of the periods 

62 of time. 

1 58. The computing device of claim 57 wherein the trace information 

2 receiver component, trace information analysis component and presentation component 

3 are executing in memory of the computing device. 

1 59. A computer-readable medium whose contents cause a computing 



2 device to analyze execution of multiple threads of a software program on a first 

■p 3 computer, the first computer having multiple processors that each have multiple 

it 4 protection domains that are each able to execute at least one of the multiple threads, each 

JS 5 processor having a coimter indicating a number of instruction holes during which an 

: 6 instruction is not executed by the processor, each protection domain having a counter 

5^ 7 indicating a number of instructions issued in the protection domain by all executing 

;i 8 threads, by: 



,t 9 receiving an indication of trace information reflecting a series of events that 

= : 10 occurred during the execution, each event associated with execution of one of the 

L-i 11 multiple threads by one of the protection domains of one of the processors and each event 

12 having associated values ui the trace information of variables maintained by the executing 

13 software program, by the one protection domain, and/or by the one processor; 

14 for each of a plurality of periods of time during which the execution was 

15 occurring, 

16 determining from the trace information a number of instructions 

17 executed for the software program during the period of time by 

18 identifying multiple protection domains that each executed at 

19 least one of the multiple threads during at least a portion of the period of time; 

20 for each of the identified protection domains, 

21 determining a change in the value of the issued 
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22 instructions counter of the protection domain during the period of time; 

23 determining if all of the instructions issued in the 

24 protection domain during the period of time were for one of the multiple threads; 

25 when it is determined that all of the instructions issued 

26 in the protection domain during the period of time were for one of the multiple threads, 

27 calculating a value for the number of instructions executed for the software program 

28 during the period of time by the protection domain to be the determined change; and 

29 when it is determined that all of the instructions issued 

30 in the protection domain during the period of time were not for one of the multiple 
331 threads, calculating a value for the number of instructions executed for the software 
i~ 32 program during the period of time by the protection domain to be a portion of the 
:r 33 determined change that corresponds to a portion of the period of time during which at 
7 34 least one thread for the software program was executing in the protection domain; and 
?35 determining the number of instructions executed for the 
f.. 36 software program during the period of time to be a sum of the calculated values for each 
iL 37 of the identified protection domains; and 

'2 38 determining from the trace information a mmiber of instruction slots 

39 available for execution of the instructions of software program during the period of time 

40 by 

41 identifying processors that each executed at least one of the 

42 multiple threads during the period of time; 

43 for each of the identified processors, 

44 determining a change in the value of the instruction 

45 holes counter of the processor during the period of time; and 

46 if all of the instruction holes that occurred during the 

47 period of time were attributable to the software program, calculating a value for the 

48 number of instruction holes for the processor that are attributable to the software program 

49 during the period of time to be the determined change in the value of the instruction holes 
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50 counter; 

51 calculating a value for the number of instruction holes that are 

52 attributable to the software program during the period of time by all of the identified 

53 processors to be a sum of the calculated values for each of the identified processors; and 

54 determining the number of instruction slots available for 

55 execution of the instructions of software program during the period of time to be a sum of 

56 the determined number of instructions executed for the software program during the 

57 period of time and of the calculated value for the number of instruction holes that are 

58 attributable to the software program during the period of time; and 

59 presenting to a user an indication of the determined number of executed 

60 instructions for each of the periods of time and an indication of the determined number of 

61 available instruction slots for each of the periods of time. 

1 60. The computer-readable medium of claim 59 wherein the computer- 

2 readable medium is a memory of a computer. 

1 61. The computer-readable medium of claim 59 wherein the computer- 

2 readable medium is a transmission medium containing generated data that includes the 

3 contents. 

1 62. A method for generating trace iirformation reflecting a series of 

2 events that occurred during execution of multiple software threads on a first computer, 

3 the method comprising: 

4 receiving an indication of a software program for which trace information is 

5 to be generated; 

6 receiving indications from a user of one or more locations within the 

7 software program at which trace information is to be generated and of one or more types 

8 of event each indicating distinct types of trace information; and 
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9 



automatically producing an executable version corresponding to the 



10 software program such that when executed the produced executable version will generate 

11 trace information corresponding to multiple events of the types indicated by the user, the 

12 producing of the executable version including adding multiple groups of instructions to 

13 the software program at the locations specified by the user, each of the added groups of 

14 instructions corresponding to one of the types of events indicated by the user such that 

15 when executed that added group of instructions will generate the distinct types of trace 

16 information for that type of event, the produced executable version having one or more 

17 portions which caa be executed in parallel with multiple software threads, 

18 so that execution of the produced executable version will generate trace information 

19 reflecting a series of events that occurred during execution. 

1 63 . The method of claim 62 including executing the produced executable 

2 version to generate the trace information. 

1 64. The method of claim 62 wherein the software program is a source 

2 code program, and wherein the producing of the executable version includes compiling 

3 the source code program. 

1 65. The method of claim 64 wherein the multiple groups of instructions 

2 are added to the software program before the compiling. 

1 66. The method of claim 62 wherein the producing of the executable 

2 version includes adding groups of instructions at a plurality of automatically identified 

3 locations of the software, the added groups of instructions to each generate trace 

4 information of a specified type when executed. 
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67. The method of claim 66 wherein the automatically identified 
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2 locations include beginnings of functions having more lines than a threshold. 

1 68. The method of claim 66 wherein the automatically identified 

2 locations include ends of functions having more lines than a threshold. 

1 69. The method of claim 66 wherein the automatically identified 

2 locations include locations where a compiler adds instructions related to parallelizing the 

3 execution of the executable version. 

1 70. The method of claim 69 wherein the instructions related to 

2 parallelizing the execution of the executable version are instructions related to a fork. 

1 71. The method of claim 69 wherein the instructions related to 

2 parallelizing the execution of the executable version are instructions related to a join. 

1 72. The method of claim 62 wherein, for at least some of the groups of 

2 instructions added to the executable program, additional instructions are added to the 

3 executable program that when executed creates a descriptor object for that group of 

4 instructions. 

1 73. The method of claim 72 wherein the additional groups of 

2 instructions are added in such a manner as to be executed before any of the added groups 

3 of instructions. 



1 

2 

3 



74. The method of claim 62 wherein the first computer has multiple 
processors that each have multiple protection domains each able to execute at least one of 
the multiple threads. 
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1 75 . The method of claim 62 wherein the first computer provides multiple 

2 sources of information related to hardware aspects of the first computer that vary with 

3 execution of programs on the fnst computer, and wherein the distinct types of trace 

4 information for some of the types of events includes at least one of the sources of 

5 information related to the hardware aspects. 

1 76. The method of claim 62 wherein the first computer provides multiple 

2 sources of information related to software properties of the first computer that vary with 

3 execution of programs on the fnst computer, and wherein the distinct types of trace 

4 information for some of the types of events includes at least one of the sources of 

5 information related to the software properties. 



1 77. A computer-readable mediimi containing instructions that when 

2 executed cause a computing device to generate trace information reflecting a series of 

3 events that occurred during execution of multiple software threads on a first computer, 

4 by: 

5 receiving an indication of a software program for which trace information is 

6 to be generated; 

7 receiving indications of one or more locations within the software program 

8 at which trace information is to be generated and of one or more types of event each 

9 indicating distinct types of tiace information to be generated; and 

10 automatically producing an executable version corresponding to the 

11 software program such that when executed the produced executable version will generate 

12 trace information corresponding to multiple events of the indicated types, the producing 

13 of the executable version including adding multiple groups of instructions to the software 

14 program at the locations specified by the user, each of the added groups of instructions 

15 corresponding to one of the indicated types of events such that when executed that added 
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16 group of instructions will generate the distinct types of trace information for that type of 

17 event, the produced executable version having one or more portions which can be 

18 executed in parallel with multiple software threads. 



1 78. A computing device for generating trace information reflecting a 

2 series of events that occurred during execution of multiple software threads on a fnst 

3 computer, comprising: 

4 an input receiver component capable of receiving an indication of a 

5 software program for which trace information is to be generated, and of receiving 

6 indications from a user of one or more locations within the software program at which 

7 trace information is to be generated and of one or more types of event each indicating 

8 distinct types of trace information; and 

9 an executable generator component capable of producing an executable 



10 version corresponding to the software program such that when executed the produced 

11 executable version will generate trace information corresponding to multiple events of the 

12 types indicated by the user, the producing of the executable version including adding 

13 multiple groups of instructions to the software program at the locations specified by the 

14 user, each of the added groups of instructions corresponding to one of the types of events 

15 indicated by the user such that when executed that added group of instructions will 

16 generate the distinct types of trace information for that type of event, the produced 

17 executable version having one or more portions which can be executed in parallel with 

18 multiple software threads. 



1 79. A method for generating trace information reflecting a series of 

2 events that occurred during execution of multiple software threads on a first computer, 

3 the method comprising: 

4 receiving an indication of an executable software program that when 

5 executed will generate trace information corresponding to multiple events of specified 
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6 types, the executable software program including multiple groups of instructions added at 

7 specified locations and each corresponding to one of the specified types of events; and 

g generating the trace information by executing the executable software 

9 program using multiple software threads on the first computer in such a manner that each 

10 of the added group of instructions are executed at least once, each execution of an added 

11 group of instructions corresponding to a specified type of event generating trace 

12 information of the type for that type of event. 

1 80. The method of claim 79 including, before the generating of the trace 
: : 2 information by the executing of the executable software, producing the executable 
!i; 3 software program from software source code in such a manner that the multiple groups of 
I 4 instructions are added at the specified locations. 

Ml 81. The method of clarni 79 wherein the specifications of the locations 

n 2 and of the types of events for the multiple added groups of instructions are received firom 

X 3 a user. 

Vi 1 82. The method of claim 79 wherein, during the executing of the 

2 executable software program, the execution of each of the added groups of instructions 

3 adds current values of one or more variables to the generated trace information, the one or 

4 more variables maintained by tbe executing software program and/or by the first 

5 computer. 

1 83. The method of claim 79 wherein the first computer has multiple 

2 processors that each have multiple protection domains each able to execute at least one of 

3 the multiple threads. 



1 



84. The metiiod of claim 79 wherein, during the executing of the 
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2 executable software program and before the execution of an added group of instructions, 

3 an additional group of added instructions is executed and creates a descriptor object for 

4 that added group of instructions. 

1 85. The method of claims 79 wherein information is added to the 

2 generated indicated trace information by an executing program other than the executable 

3 software program. 

1 86. The method of claim 85 wherein the other executing program is au 

2 operating system. 

1 87. The method of claim 85 wherein the other executing program is a 

2 background daemon. 

1 88. A computer-readable medium whose contents cause a computing 

2 device to generate trace information reflecting a series of events tiiat occurred during 

3 execution of multiple software threads on a first computer, by: 

4 receiving an indication of an executable software program that when 

5 executed will generate trace information corresponding to multiple events of specified 

6 types, the executable software program including multiple groups of instiiictions added at 

7 specified locations and each corresponding to one of the specified types of events; and 

8 generating the ti:ace information by executing the executable software 

9 program using multiple software threads on the first computer in such a manner tiiat each 

10 of the added group of instiiictions are executed at least once, each execution of an added 

11 group of instructions corresponding to a specified type of event generating trace 

12 information of the type for that type of event. 



1 



89. A computing device for generating trace information reflecting a 
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2 series of events that occurred during execution of multiple software threads on a first 

3 computer, comprising: 

4 an input receiver component capable of receiving an indication of an 

5 executable software program that when executed will generate trace information 

6 corresponding to multiple events of specified types, the executable software program 

7 including multiple groups of instructions added at specified locations and each 

8 corresponding to one of the specified types of events; and 

9 a trace information generator component capable of generating the trace 

10 information by executing the executable software program using multiple software 

11 threads on the first computer in such a manner that each of the added group of 

12 instructions are executed at least once, each execution of an added group of instructions 

13 corresponding to a specified type of event generating trace information of the type for 

14 that type of event. 



1 90. A method for analyzing trace information generated during 

2 execution of multiple threads of a software program on a fnst computer, the generated 

3 trace information reflecting a series of events that occurred durmg the execution, the 

4 method comprising: 

5 receiving an indication of trace information generated during execution of 



6 an executable software program using multiple software threads on the first computer, the 

7 executable software program mcludmg multiple groups of instructions each 

8 corresponding to a specified type of event, the execution such that each of the multiple 

9 group of instructions are executed at least once and such that each execution of an added 

10 group of instructions generates trace information of a type corresponding to the specified 

1 1 type of event for that added group of instructions; and 

12 analyzing the generated trace information to extract execution mformation 

13 corresponding to the events that occurred during the execution 
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1 9 1 . The method of claim 90 including producing the executable software 

2 program from software source code in such a manner that the multiple groups of 

3 instructions are added at the specified locations. 

1 92. The method of claim 90 including generating the indicated trace 

2 information by executing the executable software program using multiple software 

3 threads on the first computer. 

1 93. The method of claim 90 wherein, during the executing of tiie 

2 executable software program, the execution of each of the added groups of instiictions 

3 adds current values of one or more variables to tiie generated trace information, the one or 

4 more variables maintained by the executing software program and/or by tiie first 

5 computer. 

1 94. The method of claim 90 wherein the first computer has multiple 

2 processors tiiat each have multiple protection domains each able to execute at least one of 

3 the multiple threads. 

1 95 . The method of claim 90 wherein the analyzing of tiie generated trace 

2 information includes using a description of the specified types of events to identify 

3 groups of trace information each reflecting execution of a group of added instiiictions 

4 corresponding to tiiose specified types of events. 

1 96. The metiiod of claim 95 wherein the analyzing of tiie tiace 

2 information includes, when information for an event in tiie generated trace information is 

3 separated in multiple non-contiguous portions, creating a mapping to assist in retrieving 

4 the information for the event. 
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1 97. The method of claim 96 including using the identified groups of 

2 trace information and the created mapping to extract current values for execution 

3 information. 

1 98. The method of claim 90 wherein the analyzing of the trace 

2 information includes modifying current values of events so as to be normalized with 

3 respect to beginning of the execution. 

1 99. The method of claim 90 wherein the analyzing of the trace 

2 information includes modifying current values of events so that the values reflect only the 

3 execution of the multiple threads. 

1 100. The method of claim 90 including using information retrieved from 

2 created descriptor objects in the generated trace information. 

1 10 L A computer-readable medium whose contents cause a computing 

2 device to analyze trace information generated during execution of multiple threads of a 

3 software program on a first computer, the generated trace information reflecting a series 

4 of events that occurred during the execution, by: 

5 receiving an indication of trace information generated during execution of 



6 an executable software program using multiple software threads on the fnst computer, the 

7 executable software program including multiple groups of instructions each 

8 corresponding to a specified type of event, the execution such that each of the multiple 

9 group of instructions are executed at least once and such that each execution of an added 

10 group of instructions generates trace information of a type corresponding to the specified 

11 type of event for that added group of instructions; and 

12 analyzing the generated trace information to extract execution information 
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13 corresponding to the events that occurred during the execution 

1 102. A computing device for analyzing trace information generated during 

2 execution of multiple threads of a software program on a first computer, the generated 

3 trace information reflecting a series of events that occurred during the execution, 

4 comprising: 

5 an input receiver component capable of receiving an indication of trace 

6 information generated durmg execution of an executable software program using multiple 

7 software threads on the furst computer, the executable software program including 

8 multiple groups of instructions each corresponding to a specified type of event, the 

9 execution such that each of the multiple group of instructions are executed at least once 

10 and such that each execution of an added group of instructions generates trace 

11 information of a type corresponding to the specified type of event for that added group of 

12 instructions; and 

13 a trace information analysis component capable of analyzing the generated 

14 trace information to extract execution information corresponding to the events that 

15 occurred during the execution. 



1 103. A method for analyzing trace information generated during 

2 execution of multiple threads of a software program on a first computer, the generated 

3 trace information reflecting a series of events that occurred during the execution, the 

4 method comprising: 

5 receiving an indication of execution information extracted from trace 

6 information generated during execution of an executable software program using multiple 

7 software threads on the first computer, the execution information corresponding to events 

8 that occiirred during the execution, the executable software program includhig multiple 

9 groups of instructions each corresponding to a specified type of event, the execution such 
10 that each of the multiple group of instructions are executed at least once and such that 
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11 each execution of an added group of instructions generates trace information of a type 

12 corresponding to the specified type of event for that added group of instructions; and 

13 for each of muhiple periods of time, presenting to a user an indication of 

14 the extracted execution information for that period of time. 

1 104. The method of claim 103 including analyzing the generated trace 

2 information to extract execution information. 



1 105. The method of claim 103 wherein the presenting to the user includes 

2 displaying a graph including the indications. 

1 106. The method of claim 103 wherein the indicated extracted execution 

2 information includes an indication of a determined number of executed instructions for 

3 each of the periods of time and an indication of a determined number of available 

4 instruction slots for each of the periods of time. 

1 107. The method of claim 106 wherein the displayed indication of the 

2 determined number of available instruction slots for each period of time includes a 

3 displayed indication of a calculated number of instruction holes that are attributable to the 

4 executing software program during the period of time, with the calculated mmiber of 

5 instruction holes displayed in such a mamier that a user can visually aggregate the 

6 displayed indication of the determined number of executed instructions for that period of 

7 time with the displayed indication of calculated number of instruction holes for that 

8 period of time. 

1 108. The method of claim 106 wherein the displayed graph includes a 

2 time-based axis, and wherein the displayed indications of the determined number of 

3 executed instructions and the determined number of available instruction slots for each of 
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4 the periods of time are points on the graph. 

1 109. The method of claim 108 wherein the displayed graph includes an 

2 origin with at least two axes, and including, after the displaying of the indications of the 

3 determined number of executed instructions and of the determined number of available 

4 instruction slots, redefining at least one of the axes based on a new indicated displayed 

5 location. 



1 110. The method of claim 103 mcluding, for at least one of the periods of 

2 time, presenting an indication of a logical code block of the executable software program 

3 that was executing during that period of time. 

1 111. The method of claim 110 wherein the presented indication of the 

2 logical code block is a name of the logical code block. 

1 112. The method of claim 110 wherem the presented indication of the 

2 logical code block is source code of the logical code block. 

1 113. The method of claim 110 wherein the logical code block is a 

2 function. 

1 114. The method of claim 103 wherein the extracted execution 

2 information includes values of one or more variables maintained by the executing 

3 software program and/or by the first computer, and including presenting at least some of 

4 the included variable values in a tabular format, 

1 115. The method of claim 103 whereui a number of processors of the first 

2 computer that are identified during each of the periods of time is greater than one, and 
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3 wherein a determined number of available instruction slots for each of the periods of time 

4 is the identified number of processors for that period of time. 

1 116. The method of claim 1 03 wherein an identified number of processors 

2 of the first computer for at least one of the periods of time is greater than one, and 

3 wherein information for each of the processors is aggregated during the presenting of 

4 information for those periods of time. 

1 117. The method of claim 116 including normalizing the information for 



2 each of the processors prior to the presenting of the information by determining a value 

3 for an instruction holes counter of each of the processors at a beginning of the execution 

4 of the software program and by subtracting the determined value for each instruction hole 

5 counter from each later value of that instruction hole counter. 



1 118. The method of claim 103 wherein information to be presented is 

2 determined based on a specification provided by a user. 

1 119. The method of claim 118 wherein the specification is used at the 

2 time of the presenting to determine the information to be presented. 

1 120. The method of claim 103 including presenting information about 

2 memory references performed during at least one of the periods of time. 

1 121. The method of claim 103 uicluding presenting information about 

2 FLOPS performed during at least one of the periods of time. 

1 122. The method of claim 103 including presenting information about a 

2 number of the threads in existence during at least one of the periods of time. 
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1 123. The method of claim 103 including presenting infonnation about a 

2 number of the threads that are blocked during at least one of the periods of time. 

1 124. The method of claim 103 includmg presenting information about a 

2 number of the threads that are ready for execution during at least one of the periods of 

3 time. 

1 125. The method of claim 103 including presenting information about 

2 contention for locks during at least one of the periods of time. 

1 126. The method of claim 103 wherein the presenting of information is in 

2 response to a request by a user. 

1 127. The method of claim 103 wherein the presenting of information is 

2 performed automatically without user intervention. 

1 128. A computer-readable mediimi whose contents cause a computing 

2 device to analyze trace information generated during execution of multiple threads of a 

3 software program on a first computer, the generated trace information reflecting a series 

4 of events that occurred durnig the execution, by: 

5 receiving an indication of execution information extracted from trace 



6 information generated during execution of an executable software program using multiple 

7 software threads on the first computer, the execution information corresponding to events 

8 that occurred during the execution, the executable software program including multiple 

9 groups of instructions each corresponding to a specified type of event, the execution such 

10 that each of the multiple group of instructions are executed at least once and such that 

11 each execution of an added group of instructions generates trace information of a type 
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12 corresponding to the specified type of event for that added group of instructions; and 

13 for each of muhiple periods of time, presenting to a user an indication of 

14 the extracted execution information for that period of time. 

1 129. A computing device for analyzing trace information generated during 

2 execution of multiple threads of a software program on a first computer, the generated 

3 trace information reflecting a series of events that occurred during the execution, 

4 comprising: 

5 an input receiver component capable of receiving an indication of execution 

6 information extracted from trace information generated during execution of an executable 

7 software program using multiple software threads on the first computer, the execution 

8 information corresponding to events that occurred during the execution, the executable 

9 software program including multiple groups of instructions each correspondmg to a 

10 specified type of event, the execution such that each of the multiple group of instructions 

11 are executed at least once and such that each execution of an added group of instructions 

12 generates trace information of a type corresponding to the specified type of event for that 

13 added group of instructions; and 

14 a trace information presenter component capable of, for each of multiple 

15 periods of time, presenting to a user an indication of the extracted execution information 

16 for that period of time. 

17 



